Automatic keystone correction for projectors with arbitrary orientation

ABSTRACT

A method corrects keystoning in a projector arbitrarily oriented with respect to a display surface. An elevation angle, a roll angle, and an azimuth angle of an optical axis of the projector are measured with respect to the display surface. A planar projective transformation matrix is determined from the elevation, roll, and azimuth angles. A source image to be projected by the projector is warped according to the planar projective transformation, and then projected onto the display surface.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] A claim of priority is made to U.S. Provisional PatentApplication Ser. No. 60/226,066, filed Aug. 17, 2000, entitled AUTOMATICKEYSTONE CORRECTION FOR PROJECTORS WITH ARBITRARY ORIENTATION.

FIELD OF THE INVENTION

[0002] This invention relates generally to image projectors, and moreparticularly, to correcting keystoning problems in projected images.

BACKGROUND OF THE INVENTION

[0003] Portable digital image projectors are common. These projectors,while connected to a PC or VCR, sit on a table and are aimed at aprojection surface to show a “slide” presentation or a video. Many ofthese projectors use transmission LCDs, and typically only have a singlemain lens. The projectors can display images one at the time or as asequence of images.

[0004] These projectors are typically designed so that level undistortedimages are projected on the projection surface when the projector isplaced horizontally on a level support surface, so that the projector'soptical axis is lined up perpendicular to the, typically, verticalprojection surface. If any of the above assumptions is violated, thenthe resulting image on the projection surface may not be rectangular andwill be, at best, a trapezoid, and at worst an arbitrary quadrilateral.This problem is called keystoning.

[0005] With prior art projectors, the only way to correct keystoning isto tediously adjust the physical position of the projector by moving itaround, tilting and rotating it, until a near rectangular image isdisplayed. In some cases, it may be impossible to physically adjust theposition of the projector. For example, the projector may need to bewell above or below the display surface. Some prior art projectorscorrect the distorted image optically or by the user providing projectorpositional data.

[0006] U.S. Pat. No. 5,548,357, “Keystoning and focus correction for anoverhead projector,” issued to Appel et. al. on Aug. 18, 1998, describesa system where a test slide is displayed. A user identifies line pairsmost parallel to each other. The line pair identification activates adistortion correction program that uses the oblique angle between thehorizontal plane through the projector and the viewing screen.

[0007] U.S. Pat. No. 5,795,046, “Method for pre-compensating anasymmetrical picture in a projection system for displaying a picture,”issued to Woo on August 1998, describes a system where the projectionangle, and the trapezoidal error, is compensated for by the userentering in positional information into the system via a keyboard.

SUMMARY OF THE INVENTION

[0008] A method corrects keystoning in a projector arbitrarily orientedwith respect to a display surface. An elevation angle, a roll angle, andan azimuth angle of an optical axis of the projector are measured withrespect to the display surface.

[0009] A planar projective transformation matrix is determined from theelevation, roll, and azimuth angles. A source image to be projected bythe projector is warped according to the planar projectivetransformation, and then projected onto the display surface.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010]FIG. 1 a is a illustration of a projector oriented perpendicularlyto a planar display surface;

[0011]FIG. 1b is a illustration of a projector oriented obliquely to theplanar display surface;

[0012]FIG. 2 is a illustration of a projector according to theinvention;

[0013]FIGS. 3a-c illustrate three rotational angles associated withprojectors;

[0014]FIG. 4 is a flow diagram of a warping function used by theinvention. 4 is a schematic illustration of the image synthesis of thedisplay system

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0015]FIG. 1a show a projector 100 placed on a table 10 and oriented ata display surface 101 so that a projected image 102 is perfectlyrectangular. This requires that the optical axis of the projector 100 beperfectly aligned perpendicularly to the plane of the display surface101, and that there is no rotation about the optical axis.

[0016]FIG. 1b shows a projector 103 placed obliquely on the table 10 andaimed at the display surface 101. In this case, a projected image 104 issome arbitrary shaped quadrilateral due to the oblique angle between theoptical axis and the display surface.

[0017] The present invention describes a method and system to correctthe distorted image 104 to the required rectangular shape 102 as seen inFIG. 1. As a result, this allows a user to casually place a projector ona surface, without being concerned if the table top is perfectlyhorizontal, and the projectors optical axis is perfect perpendicular tothe display surface.

[0018] As shown in FIGS. 3a-3 b, when the projector is placed obliquely,or the table is titled, or the floor is tilted, or any combinations ofthese, the obliqueness can be expressed in terms of the three angles,elevation 301, roll 303, and azimuth 305. Each angle is a measurementbetween the actual angle of the optical axis 311 and the ideal angle 312of the optical axis, i.e., perpendicular to the display surface and norotation. In the ideal setup, as shown in FIG. 1, all the three anglesare zero.

[0019] In the present invention, the values of these three angles301-303 are determined automatically using sensors mounted on theprojector 200 as shown in FIG. 2. FIG. 2 shows the projector 200according to the invention. The projector 200 includes multiple sensors,e.g., tilt sensor 201-202, and a camera 203. The sensors 201-202 canalso be accelerometer implemented using 2D-accelerometer boards, forexample, the ADXL-202 from Analog Devices, Inc. The camera 203 can beany digital camera with digital out-put signals. The camera 203 acquiresan input image of a registration image having a predetermined pattern,e.g., a checkerboard pattern. These are well known.

[0020] It should be noted that the angles 301-303 shown in FIG. 3 can besensed by other techniques such as lasers, magnetic sensors, or gyrosensors. With fewer sensors, e.g., one or two, some keystoning effectscan be compensated. For example, a single sensor can correct for anon-zero elevation angle 301.

[0021] The sensors 201-203 are coupled to a processor 210. The processor210 is conventional in terms of its hardware and operating systemsoftware. Application software implements the method 211 according tothe invention. The method 4001 warps images, before they are projected,according to measurements taken by the sensors 201-203. The warpingcauses the otherwise arbitrary quadrilateral 104 to be projected appearas a rectangle.

[0022] The elevation angle 301 and the roll angle 320 are independent ofthe geometric relationship between the projector 200 and the displaysurface 101. Hence, these two angles can be sensed or measured usingtilt or gravity sensors. For example, the sensor 201 is placed parallelto the optical axis of the projector, and the sensor 202 that measuresthe roll angle 302 around the optical axis it is placed perpendicular tothe optical axis of the projector. The azimuth angle 303 is determinedby viewing a projected image with the camera 203.

[0023] The azimuth angle 303 is derived from a single planar projectivetransformation (homography) between pixel locations in the projectedimage and corresponding pixel locations in an input image acquired bythe camera 203 of the projected image, described in greater detailbelow.

[0024]FIG. 4 shows the steps of the warping 400 according to theinvention. Typically, the computer application 211 generates arectangular image 400, having corners generally marked A, B, C, and D.The method 400 determines a suitable quadrilateral 401, which whenprojected onto the display surface 1091, appears as a correct displayedimage 402, i.e., the image is rectangular and axis aligned with the realworld sense of what is horizontal and vertical.

[0025] The warping 400 scan converts the image 400 into thequadrilateral (warped) image 401 using the homography determined fromthe sensor measurements. The warping can be performed by texture mappingor image warping such as is available in conventional rendering engines.The homography is defined by the three angles 301-303. The warped image401 is the input for the obliquely positioned projector 200 according tothe invention. The correct displayed image 402 is an axis alignedrectangle that fits inside the quadrilateral 104. In the shown example,the axes are horizontal and vertical directions.

[0026] Rendering Process Given Three Angles Using 3D Graphics

[0027] The elevation and roll angles 301-302 are determined directlyfrom the sensor 201-202. The azimuth angle 303 is determined by thefollowing steps. Using the calibrated camera-projector pair, find thelocation of features in the input image acquired by the camera 203 ofthe registration image, e.g., corners or lines of the checkerboardpattern. The features are located in 3D in projector coordinate system.The projector coordinate system has a center of projection at theoptical center of the projector, and an optical axis along the positivez-axis of the projector coordinate system.

[0028] The equation of a plane passing through at least three featuresis determined. The normal of this plane is N=[Nx, Ny, Nz]. A projectionin the x-y plane is N_(xy)=[Nx, Ny, 0]. The dot product of a normalizedN_(xy) and projector optical axis [0 0 1] is the cosine of the azimuthangle=cos⁻¹(N_(xy).[0 0 1]). After the three angles are determined,there are a number of ways of warping and rendering a correct image.Here, is a preferred method.

[0029] The projector illuminates an image with width W and height H at adistance D from the projector. This situation can be represented by a 3Dcoordinate system with the projector is at the origin and theilluminated rectangle on the display surface 101 is parallel to the x-yplane at z=D.

[0030] The extents of the projected quadrilateral are [W_(Left),W_(Right)] along x-dimension, and [H_(Bottom), H_(Top)] along they-dimension, W=|W_(Right) -W_(Left)| and H=|H_(Top)−H_(Bottom)|. Thistype of projection cone can be represented by a 3×3 perspectiveprojection matrix P indicating the internal parameters of the projectorthat define the focal length and image center.

[0031] The three angles elevation, roll, and azimuth correspond to three3×3 rotation matrices R_(e), R_(r), R_(a). The goal is to render thesource image texture mapped on a rectangle of size W×H. The rectangle isaxis aligned in the x-y plane, and centered at the origin. The image iscalculated from a rendering camera, with parameters P and thetransformation defined below. This will give the required warpingeffect. A point [X, Y, 0] on the rectangle in x-y plane is mapped to apixel [x,y] with the following transformation and projection equation:

[x y 1]˜=P* [(R _(r) *R _(e) *R _(a) *[X Y 0]^(T))+[xs, ys, D] ^(T),

[0032] where xs=(W_(Left)+W_(Right))/2, and ys=(H_(Bottom)+H_(Top))/2.

[0033] A second method achieves the same effect by warping the sourceimage with a 3×3 homography matrix H between the corresponding set ofpoints { in } and { n}.

[0034] If

[nx ny 1]^(T) ˜=P* [(R _(r) *R _(e) *R _(a)*([(mx−xs)(my−ys)0]^(T))+[xs,ys, D] ^(T)),

[0035] then the homography H is given by

[nx ny 1]^(T) ˜=H*[mx my 0]^(T).

[0036] Four or more pairs of {m=(mx, my)} and {n=(nx, ny)} aresufficient to determine the homography matrix H. A good set of choicesfor candidate {m} used in homography computation are the corners of theimage rectangle WxH.

[0037] The invention effectively enables casual placement of a projectorto generate correct imagery without distortion eliminating expensiveelectromechanical adjustments. The invention makes it possible toinstall a projector in any orientation e.g. upside down or resting onits side, and detect the parameters of correct imagery by adding sensorsalong various directions.

[0038] This invention is described using specific terms and examples. Itis to be understood that various other adaptations and modifications maybe made within the spirit and scope of the invention. Therefore, it isthe object of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of the invention.

I claim:
 1. A method for correcting keystoning in a projectorarbitrarily oriented with respect to a display surface, comprising;measuring an elevation angle, a roll angle, and an azimuth angle of anoptical axis of the projector with respect to the display surface;determining a planar projective transformation matrix from theelevation, roll, and azimuth angles; warping a source image to beprojected by the projector according to the planar projectivetransformation; and projecting the warped source image onto the displaysurface.
 2. The method of claim 1 wherein the elevation and toll angelsare measured by tilt sensors, and the azimuth angle is a camera coupledto the projector.
 3. The method of claim 2, further comprising:acquiring an input image of a registration image; and determining theazimuth angle from the input image.
 4. The method of claim 1 furthercomprising: measuring the elevation angle with a tilt sensor.
 5. Themethod of claim 1 further comprising: measuring the roll elevation anglewith a tilt sensor.
 6. The method of claim 1 further comprising:measuring the azimuth angle with a camera-projector pair
 7. The methodof claim 1 wherein the projector uses a laser beam.
 8. A method systemfor correcting keystoning in a projector arbitrarily oriented withrespect to a display surface, comprising; a first tilt sensor measuringan elevation angle of an optical axis of the projector with respect tothe display surface; a second tilt sensor measuring an elevation angleof an optical axis of the projector with respect to the display surface;a camera coupled to the projector measuring an azimuth angle of anoptical axis of the projector with respect to the display surface; meansfor determining a planar projective transformation matrix from theelevation, roll, and azimuth angles; means for warping a source image tobe projected by the projector according to the planar projectivetransformation and projecting the warped source image onto the displaysurface with the projector.